SMSTC (2007/08)

Probability

                www.smstc.ac.uk

Contents

3 Random variables 3–1

3.1 Random variables and their distributions . . . . . . . . . . . . . . . . . . . . . . 3–1 3.1.1 Review of definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–1 3.1.2 Law (or distribution) of a random element (or variable) . . . . . . . . . . 3–3 3.1.3 Law of a discrete random variable . . . . . . . . . . . . . . . . . . . . . . 3–4 3.1.4 Uniform random variable and continuous random variables . . . . . . . . 3–4

3.2 Distribution functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–5

3.2.1 Types of distribution functions . . . . . . . . . . . . . . . . . . . . . . . . 3–5 3.3 Transformation rules and densities . . . . . . . . . . . . . . . . . . . . . . . . . . 3–7 3.4 Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–9

3.4.1 Substitution rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–10 3.5 Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–12

3.5.1 Markov, Chebyshev, Chernoff . . . . . . . . . . . . . . . . . . . . . . . . . 3–12 3.5.2 Jensen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–12

3.6 Moments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–13 3.7 Hölder, Minkowski and Cauchy-Bunyakowskii-Schwarz . . . . . . . . . . . . . . . 3–13 3.8 Moment generating functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–14 3.9 Characteristic functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–15

SMSTC (2007/08) Probability

Lecture 3: Random variables

a

             www.smstc.ac.uk

\underline{Contents}

3.1 Random variables and their distributions . . . . . . . . . . . . . . . . 3–1 3.1.1 Review of definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–1 3.1.2 Law (or distribution) of a random element (or variable) . . . . . . . . . 3–3 3.1.3 Law of a discrete random variable . . . . . . . . . . . . . . . . . . . . . 3–4 3.1.4 Uniform random variable and continuous random variables . . . . . . . 3–4

3.2 Distribution functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–5

3.2.1 Types of distribution functions . . . . . . . . . . . . . . . . . . . . . . . 3–5 3.3 Transformation rules and densities . . . . . . . . . . . . . . . . . . . . 3–7 3.4 Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–9

3.4.1 Substitution rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–10 3.5 Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–12

3.5.1 Markov, Chebyshev, Chernoff . . . . . . . . . . . . . . . . . . . . . . . . 3–12 3.5.2 Jensen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–12

3.6 Moments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3–13 3.7 Hölder, Minkowski and Cauchy-Bunyakowskii-Schwarz . . . . . . . . 3–13 3.8 Moment generating functions . . . . . . . . . . . . . . . . . . . . . . . 3–14 3.9 Characteristic functions . . . . . . . . . . . . . . . . . . . . . . . . . . 3–15

a

3.1 Random variables and their distributions

3.1.1 Review of definitions

Recall that a random variable is a measurable function from a measurable space (Ω, ℱ) into (ℝ, ℬ). More generally, A random element is a measurable function X between two measurable spaces, an “abstract” one, (Ω, ℱ), and a “concrete”^{b} one, (S, 𝒮). In other words, we require that the inverse image by X of each element of 𝒮 be an element of ℱ. We denote this situation by

X : (Ω, ℱ) → (S, 𝒮).

SMST C: Probability 3–2

3–1 For example, if (S, 𝒮) = (ℝ^{d}, ℬ(ℝ^{d})), where ℬ(ℝ^{d}) are the Borel sets on ℝ^{d}, we refer to X as d (real) random variables or a random vector because we may, by choosing Cartesian coordinates on ℝ^{d}, represent X by (X_{1}, …, X_{d}), where X_{1} is one random variable, X_{2} is one random variable, ..., X_{d} is one random variable..

aThese notes contain almost no proofs. For a complete set of notes, see [3]; alternatively, read the introductory book [4], or the more advanced book [1].

bThe adjectives in quotes have nothing to do with Mathematics but, rather, with our human interpretation of it.

EXERCISE 1. Show that if X : (Ω, ℱ) → (S, 𝒮), H : (S, 𝒮) → (T, 𝒯 ) are random elements then H^{∘}X is a random element.

Recall that the σ-algebra generated by a collection, say 𝒜, of subsets of Ω, is defined as the intersection of all σ-algebras containing 𝒜; it is denoted by σ(𝒜).

Lemma 3.1. If X : Ω → S is a function, ℱ a σ-algebra on Ω and 𝒮 a σ-algebra on S generated by the collection of sets 𝒞 then X is a random variable if and only if X^{−1}(B) ∈ ℱ for all B ∈ 𝒞.

Recall that the notation {X ∈ B} = X^{−1}(B) is used all the time, so we will stick to it when we can.

Corollary 3.1. If X : Ω → ℝ is a function and ℱ a σ-algebra on Ω, then X is one random variable ⟺ {X ≤ x} ∈ ℱ for all x ∈ ℝ ⟺ {X < x} ∈ ℱ for all x ∈ ℝ ⟺ {X > x} ∈ ℱ for all x ∈ ℝ.

If X_{n} is a sequence of random variables then X_{1} + X_{2}, X_{1} · X_{2}, X_{1} ∧ X_{2} are all random variables. Also, recall that inf_{j} X_{j}, \underline{lim}_{j} X_{j}, sup_{j} X_{j}, \overline{lim}_{j} X_{j} are random variables in ℝ ∪ \{+∞, −∞\}. If X : Ω → S is a function and 𝒮 a fixed σ-algebra on S, the σ-algebra generated by X is defined by

σ(X) := {X^{−1}(B), B ∈ 𝒮 }.

Let us study random elements in a product S_{1} × ⋯ × S_{d} of sets. Suppose that on each S_{i} we have a σ-algebra 𝒮_{i} ⊂ 2^{S_{i}}. We first construct a natural σ-algebra on S_{1} × ⋯ × S_{d}. For each i consider the projection function

π_{i} : S_{1} × ⋯ × S_{d} → S_{i}; π_{i} : (s_{1}, ⋯, s_{d}) ↦ s_{i}.

Define

𝒮_{1} ⊗ ⋯ ⊗ 𝒮_{d} := σ(π_{1}, …, π_{d}).

EXERCISE 2. Consider (S_{i}, 𝒮_{i}), i = 1, …, d. Let d = 2 for simplicity. Show that

𝒮_{1} ⊗ 𝒮_{2} = σ({B_{1} × S_{2} : B_{1} ∈ 𝒮_{1}} ∪ {S_{1} × B_{2} : B_{2} ∈ 𝒮_{2}}).
= σ({B_{1} × B_{2} : B_{1} ∈ 𝒮_{1}, B_{2} ∈ 𝒮_{2}}).

Lemma 3.2. Let X_{i} : (Ω, ℱ) → (S_{i}, 𝒮_{i}), i = 1, 2, …, d, be random elements. Let S = S_{1} × ⋯ × S_{d}, 𝒮 = 𝒮_{1} ⊗ ⋯ ⊗ 𝒮_{d}. Then (X_{1}, …, X_{d}) : (Ω, ℱ) → (S, 𝒮) is a random element.

EXERCISE 3. Show that, if X = 1_{A}, the indicator of a set A, then σ(1_{A}) = {∅, A, A^{c}, Ω}. Consider next two subsets A_{1}, A_{2} of Ω and prove that the σ-algebra generated by c_{1}1_{A_{1}} + c_{2}1_{A_{2}} is

σ(c_{1}1_{A_{1}} + c_{2}1_{A_{2}}) = {∅, Ω, A_{1}, A_{2}, A_{1}^{c}, A_{2}^{c}, A_{1}A_{2}, A_{1}A_{2}^{c}, A_{1}^{c}A_{2}, A_{1}^{c}A_{2}^{c},

A_{1} ∪ A_{2}, A_{1} ∪ A_{2}^{c}, A_{1}^{c} ∪ A_{2}, A_{1}^{c} ∪ A_{2}^{c}, A_{1}△A_{2}, (A_{1}△A_{2})^{c}}.

EXERCISE 4. Consider the following partition of Ω:

{A_{1}A_{2}, A_{1} \ A_{2}, A_{2} \ A_{1}, (A_{1}A_{2})^{c}}.

Show that any nonempty set in Exercise 3 can be obtained as union of elements of this partition.

EXERCISE 5. Given a partition 𝒞 = {C_{1}, …, C_{n}} of Ω show that the σ-algebra generated by 𝒞 consists of the empty set and all sets that can be obtained by taking unions of sets in 𝒞. Assuming that none of the C_{i} is empty, this 𝒞 contains exactly 2^{n} sets. Also show that any random variable that takes value b_{i} on C_{i} for each i generates a σ-algebra which is contained in σ(𝒞). Show that if the values b_{i} are distinct then σ(X) = σ(𝒞).

If X, Y are random variables on a common measurable space (Ω, ℱ) (and values in arbitrary sets) we say that Y is measurable with respect to X, often written as X ∈ σ(Y) if

σ(Y) ⊂ σ(X).

Lemma 3.3. If X : (Ω, ℱ) → (S, 𝒮), Y : (Ω, ℱ) → (ℝ, ℬ) are random variables, and if Y ∈ σ(X), then there exists a random variable (measurable map) H : (S, 𝒮) → (ℝ, ℬ) such that Y = H^{∘}X.

EXERCISE 6. Using Exercise 5 show that any Y, measurable with respect to X, must be of the form

Y = ∑_{i=1}^{n} c_{i}1(X = x_{i}).

We then see that what is claimed in Lemma 3.3 is correct, in this special situation with X being finitely-valued. Indeed, let H(x) = ∑_{i=1}^{n} c_{i}1(x = x_{i}) and, obviously, Y = H^{∘}X. The general case requires an approximation result that says that any measurable random variable X : (Ω, ℱ) → (ℝ, ℬ) can be approximated by simple random variables. A simple random variable is a random variable with finitely many values.

Lemma 3.4. Let X : (Ω, ℱ) → (ℝ, ℬ). Then there exists a sequence X_{1}, X_{2}, … of simple random variables such that lim_{n→∞} X_{n}(ω) = X(ω). If X(ω) ≥ 0, ω ∈ Ω, we can choose the sequence so that 0 ≤ X_{n}(ω) ≤ X_{n+1}(ω) for each n and ω.

The proof of Lemma 3.4 can be found in [3] and is based on defining

τ_{n}(x) := 2^{−n}⌈2^{n}x⌉ ∧ n, X_{n}(ω) := τ_{n}(X(ω)),

and the observation that X_{n} converges to X.

EXERCISE 7. Let X be one random variable in ℝ. Show that σ(X^{2}) is a strict subset of σ(X). But show that σ(2^{X}) = σ(X).

3.1.2 Law (or distribution) of a random element (or variable)

The reader will of course have noticed that the concept of “random element” has nothing to do with randomness.^{c} Let P be a probability on (Ω, ℱ). The distribution or law of the random element X is a probability P_{X} on (S, 𝒮) which is generated, in the most natural fashion, by X:

P_{X}(B) = P(X^{−1}(B)), B ∈ 𝒮 .

Note that P_{X} depends on two functions: the function P and the function X.

EXERCISE 8. Show that if (Ω, ℱ , P) is a probability space and X : (Ω, ℱ) → (S, 𝒮) a random element then (S, 𝒮 , P_{X}) is a probability space. Hence if X is a random vector then X_{1} is a random variable.

cThe name is justified because of the way the concept is used.

EXERCISE 9. With the notation of Exercise 1, show that the law of H^{∘}X as a random element on the probability space (Ω, ℱ , P) is the same as the law of H as a random element on (S, 𝒮 , P_{X}).

So the role of a random element is to transform an abstract probability space into a concrete one. In practise, one is often given^{d} a probability measure Q on some (S, 𝒮) and one may (or may not) want to construct a probability space (Ω, ℱ , P) and a random variable X : (Ω, ℱ) → (S, 𝒮) such that P_{X} = Q. In the absence of any further requirement we consider the so-called canonical construction: take Ω = S, ℱ = 𝒮, P = Q and let X(ω) ≡ ω. Then, obviously, P_{X} = Q.

3.1.3 Law of a discrete random variable

A discrete random variable X : (Ω, ℱ) → (S, 𝒮) is, by definition, one that takes countably many values. In other words, if P is a probability on (Ω, ℱ) then X is discrete if and only if there is a countable set D ∈ 𝒮 such that P(X ∈ D) = 1. We also assume that D and all its subsets are members of 𝒮. Hence the law P_{X} of X is a probability on D. We know that a probability on a countable set D can be defined by defining its values on singletons. These values form the so-called probability mass function. Thus, the probability mass function is

p(x) = P_{X}{x} = P(X = x), x ∈ D.

Clearly, if B ⊂ D then

P_{X}(B) = P_{X}( ⋃_{x∈B}{x}) = ∑_{x∈B} p(x).

So p is sufficient for computing P_{X}.

EXERCISE 10. Suppose that the random variable X takes n distinct values (i.e. X(Ω) is a set with n elements). Show that σ(X) has 2^{2^{n}} elements and describe (give a procedure for describing) them.

3.1.4 Uniform random variable and continuous random variables

A uniform random variable U in the interval [0, 1] is such that its law satisfies P_{U}([a, b]) = b − a for all 0 ≤ a ≤ b ≤ 1. To show that such a random variable exists is hard and the reader is referred to [3], for an approach based on using the probability space Ω = {0, 1}^{ℕ}. A more standard approach is to define U(x) = x, x ∈ [0, 1], where [0, 1] is endowed with the Borel σ-algebra and the Lebesgue measure P. Having shown that the latter exists, we simply observe that P_{U} is the law of a uniform random variable. Notice that if D is a countable set then

P(U ∈ D) = 0.

For example, P(U is rational number ) = 0. If F : ℝ → ℝ is a nondecreasing continuous function with lim_{x→−∞} F (x) = 0, lim_{x→∞} F (x) = 1, and if F^{−1}(u) := inf{x ∈ ℝ : F (x) > u} then X = F^{−1}(U) is a random variable that shares something in common with U, namely,

P(X ∈ D) = 0,

if D is a countable set. Such a random variable is called continuous.

dActually, one is seldom given anything. Either one derives something from some basic principles/requirements or one performs an experiment whereby measurements are collected and a probability measure is stipulated. The latter is the subject of Statistics.

3.2 Distribution functions

The distribution function of a random variable X is defined by:

F (x) := P_{X}(−∞, x], x ∈ ℝ.  (3.1)

This function is useful because:

Lemma 3.5. Knowledge of P_{X} on this class of sets only (semi-infinite intervals) implies knowledge of P_{X} on the whole of ℬ.

For a proof, see [3]. Next, here are some properties of F:

Lemma 3.6. (i) x_{1} < x_{2} ⇒ F (x_{1}) ≤ F (x_{2}), (ii) lim_{x→−∞} F (x) = 0, (iii) lim_{x→+∞} F (x) = 1, (iv) lim_{n→∞} F (x + 1/n) = F (x).

Remark: There is no good reason that we chose this class of sets, other than people have been using it by convention. For instance, we could have chosen open semi-infinite interval (−∞, x), in which case (iv) of Lemma 3.6 would be replaced by lim_{n→∞} F (x − 1/n) = F (x).

Definition 3.1. A function F : ℝ → ℝ is called distribution function iff (i) x_{1} < x_{2} ⇒ F (x_{1}) ≤ F (x_{2}), (ii) lim_{x→−∞} F (x) = 0, (iii) lim_{x→+∞} F (x) = 1, (iv) lim_{n→∞} F (x + 1/n) = F (x).

Corollary 3.2. If F is a distribution function then there exists a probability Q on (ℝ, ℬ) such that Q(−∞, x] = F (x) for all x ∈ ℝ.

Lemma 3.7. Let X be one real random variable with law P_{X} and distribution function F (x) = P_{X}(−∞, x], x ∈ ℝ. Then (i) P(X ∈ (a, b]) = F (b) − F (a), (ii) P(X ∈ (a, b)) = F (b−) − F (a), (iii) P(X ∈ [a, b]) = F (b) − F (a−), (iv) P(X = a) = F (a) − F (a−).

EXERCISE 11. Carefully justify the formulae in the proof of Lemma 3.7.

3.2.1 Types of distribution functions

We now discuss the various kinds of distribution functions on ℝ.

Discrete distribution functions

A discrete distribution function is the distribution function of a discrete random variable X with values in some countable subset S of ℝ. Assume that p(s) = P(X = s) > 0 for all s ∈ S. Such a distribution function satisfies F (s)−F (s−) > 0 for all s ∈ S. Indeed, F (s)−F (s−) = P(X = s). Also, if (a, b) is an open interval containing no points of S, then F is constant on (a, b). Indeed, if a < x < b then F (x) − F (a) = P(a < X ≤ x) = ∑_{s∈S,a<s≤x} P(X = s) = 0.

Example 3.1. Let X be a random variable such that P(X = n) = 2^{−n}, n ∈ ℕ. Then its distribution function looks like in Figure 3.1.

Example 3.2. Let X be a random variable such that for every rational number of the form m/n where m, n are integers with no common factors, we have P(X = m/n) = c2^{−(m+n)} where c is chosen so that P(X ∈ ℚ) = 1. Its distribution function is discrete because ℚ is countable. Unfortunately, I can’t draw it. (There are no intervals (a, b) containing no rational points.)

Figure 3.1: Distribution function of a discrete random variable

Figure 3.2: An absolutely continuous distribution function

Continuous distribution functions

A distribution function F is continuous if it is a continuous function, i.e. if F (x) − F (x−) = 0 for all x ∈ ℝ.

Example 3.3. Consider the random variable U with P(U ≤ u) = u for all u ∈ [0, 1]. Such a random variable exists (we constructed it). Its distribution function looks like in Figure 3.2.

Excepting the points 0, 1 we have that it is also differentiable with derivative f(u) = 1 if 0 < u < 1 and 0 otherwise.. If we arbitrarily define f(0) = f(1) = 0, we also have ∫_{−∞}^{u} f(t)dt = F (u) for all u ∈ ℝ. We like such distribution functions:

Absolutely continuous distribution functions

A distribution function F is called absolutely continuous if there exists a function f (called density of F) such that

F (x) = ∫_{−∞}^{x} f(t)dt, x ∈ ℝ.

e The density is not uniquely defined. For instance, it can be changed on a finite set and such a change will not affect the integral above. Usually, one^{f} imposes additional regularity conditions, such as continuity, resulting in uniqueness. But not all continuous distribution functions are absolutely continuous:

Singularly continuous distribution functions

A distribution function F is called singularly continuous if it is continuous but not absolutely continuous. We need to show that there are such functions.

EXERCISE 12. Consider the space (Ω = {0, 1}^{ℕ}, ℱ , P), where ℱ is the product σ-algebra, and P is such that P{ω ∈ Ω : ω_{1} = i_{1}, …, ω_{n} = i_{n}) = 2^{−n}, i_{1}, …, i_{n} ∈ {0, 1}, n ∈ ℕ. Let

V (ω) := ∑_{n=1}^{∞} \frac{2ω_{n}}{3^{n}} .

eThe integral in the display is a Lebesgue integral. For a definition, see [4]. For the time being, you may think of it as the standard Riemann integral of Integral Calculus.

f unconsciously

Show that the random variable V defined in Example 12 has a continuous but not absolutely

Figure 3.3: A continuous distribution function without density

continuous distribution function.

General distribution functions

Suppose that F, G are distribution functions. Then, for any λ ∈ (0, 1), the function λF +(1−λG) is a distribution function. (Probabilistically, if X, Y are random variables with distribution functions F, G, respectively, then we can define a new random variable Z which equals X with probability λ or Y with probability 1−λ.) So, if F is discrete and G continuous then λF +(1−λG) is neither discrete nor continuous: it is mixed. The question is: Can we exhaust all distribution functions by taking mixtures of the three types mentioned above? The answer is yes:

Theorem 3.1. Let F be a distribution function on ℝ. Then F can be uniquely written as

F = λ_{d}F_{d} + λ_{ac}F_{ac} + λ_{sc}F_{sc}

where F_{d}, F_{ac}, F_{sc} are discrete, absolutely continuous, singularly continuous distribution functions, respectively, and where the coefficients are nonnegative such that λ_{d} + λ_{ac} + λ_{sc} = 1.

The last two terms of this decomposition are known as the continuous part of F. The first two terms are known as the singular part of F. We will not prove this theorem, but refer, e.g. to [2].

Differentiation: a word of caution

The subject of densities involves the concept of a derivative of functions that are not necessarily everywhere differentiable. Mimicking the definition of a density, we will say that a function G has density g if G′(x) = g(x) for almost all x. The latter statement means that it holds true that G′(x) = g(x) for all x in some set A whose complement is small in the sense that for all ε > 0 there exist intervals I_{n}, n ∈ ℕ with lengths λ_{n}, n ∈ ℕ, such that ∑_{n} λ_{n} < ε and A^{c} ⊂ ∪_{n}I_{n}. For further elaboration on this important notion, without which the concept of a density cannot be properly understood, see [2, 3].

EXERCISE 13. Show that a continuous and piecewise differentiable function G is almost everywhere differentiable.

3.3 Transformation rules and densities

Consider a random variable X : (Ω, ℱ) → (ℝ, ℬ). Suppose P is a probability on (Ω, ℱ). We are interested in the distribution P_{X} of X. Suppose, for some reason, we don’t like it and want to change it to something else. There are two ways to do this. First, we can change the probability P and replace it by some other probability Q. Then P_{X} will be replaced by Q_{X}. Second, we can take a function H : (ℝ, ℬ) → (ℝ, ℬ) and replace X by H^{∘}X. Then P_{X} is replaced by P_{H∘X}. The two procedures are not, in general, equivalent. Suppose, for instance, that X is a discrete random variable. Then any one-to-one function H will not change the probabilities of singletons {s} such that P_{X}{s} > 0, but, merely, will rename them: {s} will be transformed to {H(s)} and its probability will remain the same. Even if H is not one-to-one, there is not much that H can do to change the probabilities. Consider, for instance, a random variable X with values 1, 2, 3 and probabilities p_{1}, p_{2}, p_{3}, respectively. Then the most a function H can do is either be one-to-one, in which case H(1), H(2), H(3) will retain the old probabilities, or map two points, say 1, 2, to a single point, with probability p_{1} + p_{2} and leave the the third intact. Thus, the types of changes in the distribution of a discrete X that can be achieved by taking a function of it are quite restricted. To really change its distribution ad libitum, we need to change the underlying probability P. For the case of absolutely continuous random variables, the story is different: a merely one-to-one function H can simultaneously change the values and the distribution in a quite general fashion.

Theorem 3.2. Let X be an absolutely continuous random variable in ℝ with density f. Let φ : ℝ → ℝ be strictly increasing differentiable function and let ψ be its inverse function. Then φ(X) a random variable with absolutely continuous distribution function and density

ψ′ · f^{∘}ψ on φ(ℝ),

and 0 elsewhere.

Proof Since φ is strictly increasing, its inverse function exists and has domain φ(ℝ). Then, the distribution function of φ(X) is, for any t ∈ φ(ℝ),

P(φ(X) ≤ t) = P(X ≤ ψ(t)) = ∫_{−∞}^{ψ(t)} f(x)dx.

By changing variable in the integral we have

∫_{−∞}^{ψ(t)} f(x)dx = ∫_{−∞}^{t} f(ψ(s))ψ′(s)ds,

where we set ψ′(s) = 0 for s ∉ φ(ℝ). From the definition of an absolutely continuous distribution function we see that, indeed, φ(X) has absolutely continuous distribution function and its density is the function inside the last integral. □

EXERCISE 14. Let U be a uniform random variable in the interval (0, 1). Find the density function of e^{e^{U}}.

Theorem 3.2 assumes that φ is strictly increasing. It is immediate to find out the formula for strictly decreasing φ. Generalising to more general functions is possible and relatively easy for random variables with values in ℝ (the story in ℝ^{d} is more complicated). For instance, we may assume that φ is piecewise differentiable. The problem becomes a problem in differential calculus and the general theorem is omitted. However, an example is due:

EXERCISE 15. Let X be a random variable with density f(x) = c(1 + x^{2})^{−1}, x ∈ ℝ. Let φ(x) = cosh(x). Find the density (and hence show that it exists) of φ(X).

3.4 Expectation

The expectation EX of one real random variable X is, if it can be defined, justified (for instance) by the Theorem (Law) of Large Numbers which will be proved at a later chapter. It is easy to define the expectation of a discrete random variable X with values in ℝ and probability mass function p(x). Let S be the set of x such that p(x) > 0. Then

EX = ∑_{x∈S} xp(x),

provided that this sum can be defined. We know from Analysis (see [2]) that the sum of positive numbers can be defined irrespective of which order we sum the numbers up. However, not all the summands above are necessarily positive. So let us consider the positive and negative terms separately and try to define

EX = ∑_{x∈S_{+}} xp(x) − ∑_{x∈S_{−}}(−x)p(x),

where S_{+} := {x ∈ S : p(x) > 0}, where S_{−} := {x ∈ S : p(x) < 0}. Each of the two sums, separately, is a sum of positive terms, hence it is well-defined. The only “problem” is that such a sum can take value +∞. If both sums are finite then EX is a finite number. If the first sum is +∞ but the second finite then EX = +∞. Similarly, if the first is finite but the second infinite, then EX = −∞. The only case where we cannot talk (cannot define) EX is when both sums are infinite. When X has absolutely continuous distribution function with density f, one can define EX similarly:

EX = ∫_{0}^{∞} xf(x)dx − ∫_{−∞}^{0}(−x)f(x)dx,

provided that not both integrals are infinity. The general case requires an approximation procedure, known as integral with respect to a measure. When X is a nonnegative random variable, there always exists a sequence ξ_{n} of nonnegative random variables, each of which takes finitely many values, such that ξ_{n}(ω) → X(ω), as n → ∞, for all ω ∈ Ω. We then define

EX = lim_{n→∞} Eξ_{n}.

It can be proved [3] that the limit exists and is independent of the approximation procedure. If X has no restriction on sign, we define

EX = EX^{+} − EX^{−},

provided that not both terms are +∞. We say that X is integrable (with respect to P) if both EX^{+} and EX^{−} are finite or, equivalently, if E|X| < ∞. (The latter follows from the identity |x| = x^{+} + x^{−}.)

Lemma 3.8 (algebraic properties of expectation). Suppose X, Y are integrable random variables on the same probability space (Ω, ℱ , P). Then:

(i) If P(A) = 0 then EX1_{A} = 0. (ii) E(cX) = cE(X) for all c ∈ ℝ. (iii) E(X + Y) = EX + EY. (iv) If P(X ≥ 0) = 1 then EX ≥ 0. If P(X ≤ Y) = 1 then EX ≤ EY.

Theorem 3.3 (monotone convergence theorem). For ANY sequence X_{n} of nonnegative random variables such that X_{n} ↑ X, we have EX_{n} ↑ EX.

Lemma 3.9 (Fatou’s lemma). For any sequence X_{n} of nonnegative random variables, E \underline{lim} X_{n} ≤ \underline{lim} EX_{n}.

If A ∈ ℱ we can define the expectation of X on A by:

E(X; A) := E(X1_{A}).

If P(A) > 0 we can define the expectation of X given A by:

E(X|A) := \frac{E(X; A)}{P(A)}.

We remark that E(X|A) is expectation with respect to the restriction P_{A} of P on A, i.e. with respect to the probability

P_{A} : ℱ → ℝ; P_{A}(B) := P(AB), B ∈ ℱ .

In other words,

E(X|A) = E_{P_{A}}X.

EXERCISE 16. Show that |EX| ≤ E|X| (whenever EX is defined).

Theorem 3.4 (Dominated Convergence Theorem). Let X_{n} be a sequence of random variables such that X(ω) = lim_{n→∞} X_{n}(ω) exists and such that |X_{n}(ω)| ≤ Y (ω) for all n and ω, and E|Y | < ∞. Then E|X_{n} − X| converges to zero.

3.4.1 Substitution rule

There is a little problem with the notation EX in that it uses the same letter regardless of the probability P. More correctly, one should write E_{P} X instead of EX or ∫_{Ω} XdP. Let us consider compositions of random elements.

Lemma 3.10. Consider the measurable functions

(Ω, ℱ) ⟶^{Z} (S, 𝒮) ⟶^{H} (ℝ, ℬ).

Let P be a probability on (Ω, ℱ). Let P_{Z} be the law of Z. Then

E_{P}H^{∘}Z = E_{P_{Z}}H,

whenever either side exists. (Here, H^{∘}Z is one real random variable on the probability space (Ω, ℱ , P) and H is one real random variable on the probability space (S, 𝒮 , P_{Z}).)

Sketch of proof: Suppose H is an indicator random variable, i.e. H = 1_{B} for some B ∈ 𝒮. Then E_{P_{Z}}1_{A} = P_{Z}(A) by the definition of the expectation of a simple random variable. On the other hand, H^{∘}Z = 1_{A}(Z) is an indicator random variable on (Ω, ℱ): it is the indicator of the set {ω ∈ Ω : Z(ω) ∈ A}. Hence, again by the by the definition of the expectation of a simple random variable, E_{P}H^{∘}Z = P(Z ∈ A). But P_{Z}(A) = P(Z ∈ A) by the definition of the law of the random variable Z (see section 3.1.3). Suppose next that H is a simple random variable. Use the above and linearity of expectation to get the result. Suppose that H is a general random variable. Use an approximation by simple random variables to conclude.

Corollary 3.3 (the expectation of a random variable depends only on its law). If X is a real random variable on (Ω, ℱ , P) with expectation E_{P}X and law P_{X} then

E_{P}X = E_{P_{X}} ι

where ι : ℝ → ℝ is the identity function: ι(x) ≡ x.

Suppose that X is a discrete random variable with values in a finite set S ⊂ ℝ and probability mass function p(x), x ∈ S. Since X = ∑_{x∈S} x1(X = x) is a simple random variable, we immediately have that EX = ∑_{x∈S} xP(X = x) = ∑_{x∈S} xp(x), as needed. The same formula holds for a discrete random variable with values in a countable set S: Simply enumerate the elements of S and use monotone convergence theorem. Let us now consider an absolutely continuous random variable X with density f. We would like to show that EX is compatible with the definition given at the beginning of the section, namely that it equals ∫_{ℝ} xf(x)dx. To do this, requires understanding of Lebesgue-Stieltjes integral. The step is omitted and can be found in [1, 4], and in [3].

Lemma 3.11. Let P_{X} be the law of a random variable X with density f. Then, for any measurable g : (ℝ, ℬ) → (ℝ, ℬ),

∫_{ℝ} g(x)f(x)dx = E_{P_{X}} g

provided that wither side exists.

If you want to see the proof, look at [3]. But do notice that, if g = 1_{B}, then this is what was discussed before the Lemma. For g simple, we use linearity. For general g we approximate.

Corollary 3.4. Let P_{X} be the law of an integrable random variable X with density f.

(i) If ι is the identity function on ℝ then

∫_{ℝ} xf(x)dx = E_{P_{X}} ι.

(ii)

EX = ∫_{ℝ} xf(x)dx.

The proof of (i) follows from Lemma 3.11 with g = ι. The proof of (ii) follows from corollary 3.3.

EXERCISE 17. Suppose that Z is a real random variable with absolutely continuous distribution function and density f_{Z}. Let H : (ℝ, ℬ) → (ℝ, ℬ) be a measurable function. Suppose that the random variable X = H(Z) has density f_{X}. Show that the expectation of X (if it exists) can be computed in two ways:

EX = ∫_{ℝ} xf_{X}(x)dx = ∫_{ℝ} H(z)f_{Z}(z)dz.

EXERCISE 18. Let B ∈ ℬ and let λ(B) := ∫_{ℝ}1_{B}(x)dx (Lebesgue integral). Show that λ : ℬ → ℝ satisfies λ(∪_{n}A_{n}) = ∑_{n} λ(A_{n}) whenever the A_{n} are mutually disjoint elements of ℬ and that λ(A + t) = λ(A) for all A ∈ ℬ, t ∈ ℝ, where A + t := {a + t : a ∈ A}. The function λ is called length.

EXERCISE 19. Consider a compass with a laser pointer attached at both ends of the needle. Suppose there is an infinite screen at some distance from the compass. Give it a spin and see mark X the location of the light with respect to a fixed point O on the screen (positive if it is to the right of O; negative if it is to the left). Show that EX is not defined. (You first must translate this problem in Mathematics.)

3.5 Inequalities

3.5.1 Markov, Chebyshev, Chernoff

Lemma 3.12 (Markov inequality). If X is a nonnegative random variable then

P(X ≥ t) ≤ \frac{EX}{t} , t > 0.

Proof We have

t1(X ≥ t) ≤ X

and E is preserved by ≤. □ Definition 3.2. The variance of a real random variable X with EX^{2} < ∞ is defined by

var X := E(X − EX)^{2}.

Lemma 3.13 (Chebyshev inequality). If X is a real random variable with EX^{2} < ∞ then

P(|X − EX| ≥ t) ≤ \frac{var X}{t}

Proof Apply the Markov inequality to |X − EX|. □ Lemma 3.14 (Chernoff inequality). If X is a real random variable then

P(X ≥ t) ≤ \frac{Eg(X)}{g(t)}

where g is a positive increasing function. Proof Since g is increasing,

{X ≥ t} ⊂ {g(X) ≥ g(t)}

Now apply the Markov inequality to g(X). □ EXERCISE 20. Let X be a discrete random variable with P(X = k) = \binom{n}{k}2^{−k}, k = 0, 1, …, n. Estimate P(X > na) for a > 0.5 using the above inequalities.

3.5.2 Jensen

A function φ : ℝ → ℝ is convex if

φ(pa + (1 − p)b) ≤ pφ(a) + (1 − p)φ(b)

for all a, b ∈ ℝ and all 0 ≤ p ≤ 1. Notice that if ξ is a random variable with P(ξ = a) = p, P(ξ = b) = 1 − p, this definition can be written as

φ(Eξ) ≤ Eφ(ξ).

Jensen’s inequality generalises this observation: Lemma 3.15. Let X be a real integrable random variable and φ a convex function. Then

φ(EX) ≤ Eφ(X).

EXERCISE 21. Let a_{1}, …, a_{n} be positive real numbers. Define their arithmetic, geometric and harmonic mean by

A_{n} = \frac{a_{1} + ⋯ + a_{n}}{n} , G_{n} = (a_{1} ⋯ a_{n})^{1/n}, H_{n} = \frac{n}{a_{1}^{−1} + ⋯ + a_{n}^{−1}} ,

respectively, and show that A_{n} ≥ G_{n} ≥ H_{n}.

3.6 Moments

Definition 3.3. When r > 0, the r-moment of a nonnegative RV X is defined as the quantity EX^{r}. The r-norm of a real RV X is defined as ||X||_{r} := (E|X|^{r})^{1/r}.

Lemma 3.16. The r-norm of X is increasing in r.

Proof Let r < s and φ(x) = x^{s/r}, x > 0. Notice that φ is convex. (This follows from the fact that its second derivative is positive.) Now apply the Jensen inequality. □

Corollary 3.5. If E|X|^{p} < ∞ for some p > 0 then E|X|^{r} < ∞ for all 0 < r < p.

3.7 Hölder, Minkowski and Cauchy-Bunyakowskii-Schwarz

Definition 3.4. If X, Y are real random variables on the same (Ω, ℱ , P), the quantity E(XY) (whenever it is defined) is called correlation between X and Y. The quantity cov(X, Y) := E((X − EX)(Y − EY)) is called covariance between X and Y.

Lemma 3.17 (Hölder inequality). Let X, Y be real random variables. Then

|E(XY)| ≤ ||X||_{p}||Y ||_{q},

for any p, q > 0, p^{−1} + q^{−1} = 1, as long as all terms involved exist and are finite.

Proof Let (Ω, ℱ , P) be a probability space on which both X, Y are defined. Without loss of generality assume that they are both nonnegative. Let q > 1 and assume that E(Y^{q}) < ∞. Consider the probability

P_{q}(A) := \frac{E(Y^{q}1_{A})}{E(Y^{q})}, A ∈ ℱ .

Let E_{q} denote expectation with respect to P_{q}. Therefore, for any nonnegative random variable W : (Ω, ℱ) → (ℝ, ℬ),

E_{q}W = \frac{E(Y^{q}W)}{E(Y^{q})}.  (3.2)

Letting, in (3.2), W = XY^{1−q}, we obtain

E(XY) = E(Y^{q}) E_{q}(XY^{1−q}).

Let p be defined from p^{−1} + q^{−1} = 1. Necessarily, p > 1. From Lemma 3.16 we have

E_{q}Z ≤ (E_{q}Z^{p})^{1/p},

for any nonnegative random variable Z : (Ω, ℱ) → (ℝ, ℬ) with EZ^{p} < ∞. Therefore,

E(XY) ≤ E(Y^{q}) (E_{q}((XY^{1−q})^{p}))^{1/p}

= E(Y^{q}) (\frac{E(X^{p}Y^{q}Y^{(1−q)p})}{E(Y^{q})})^{1/p}

= (E(Y^{q}))^{1−1/p} (E(X^{p}Y^{q+(1−q)p}))^{1/p}.

Since 1 − 1/p = 1/q and q + (1 − q)p = 0, the result follows. □

Corollary 3.6. (Cauchy-Bunyakowskii-Schwarz) Let X, Y be real random variables. Then

|E(XY)| ≤ ||X||_{2}||Y ||_{2},

as long as all terms involved exist and are finite.

Proof Notice that \frac{1}{2} + \frac{1}{2} = 1 and apply Hölder. □

Corollary 3.7. Let X, Y be real random variables. Let

ρ(X, Y) := cov(X, Y)/\sqrt{var(X)}\sqrt{var(Y)},

whenever the terms exist. Then

−1 ≤ ρ(X, Y) ≤ 1.

Lemma 3.18 (Minkowski inequality). Let X, Y be real random variables. Then

||X + Y ||_{p} ≤ ||X||_{p} + ||Y ||_{p},

for any p > 1, as long as all terms involved exist and are finite.

Proof Use the Hölder inequality as follows:

E(|X + Y |^{p}) = E(|X| |X + Y |^{p−1}) + E(|Y | |X + Y |^{p−1})
≤ [E(|X|^{p})]^{1/p} [E(|X + Y |^{(p−1)q})]^{1/q} + [E(|Y |^{p})]^{1/p} [E(|X + Y |^{(p−1)q})]^{1/q}
= (||X||_{p} + ||Y ||_{p}) [E(|X + Y |^{p})]^{1/q},
□

3.8 Moment generating functions

Let X be a real random variable. Since, for any θ ∈ ℝ, the random variable e^{θX} is nonnegative, its expectation exists (but may be equal to +∞). We define the function M : ℝ → ℝ ∪ {+∞} by

M(θ) := E(e^{θX}), θ ∈ ℝ.

Notice that M(0) = 1. This function is useful if M(θ) < ∞ for some θ ≠ 0. (Indeed, there are cases where θ = 0 is the only point at which M is finite.) If X is a positive random variable, then M(θ) < ∞ for all θ ≤ 0. If X is a negative random variable, then M(θ) < ∞ for all θ ≥ 0. M depends only on the law of X. Indeed, using the Substitution Rule (Lemma 3.10) we can write

M(θ) = E_{P_{X}} (e^{θι}), where ι(x) ≡ x,

and, if X has absolutely continuous distribution function F with density f, we can write

M(θ) = ∫_{ℝ} e^{θx}f(x)dx (Lebesgue integral) .

The function M is called moment generating function because of the following:

Lemma 3.19. Suppose there exist a < 0 < b such that M(θ) < ∞ for all a < θ < b. Then

(i) the r-moment of X exists for all r ∈ ℕ and is given by the r-derivative of M at 0:

E(X^{r}) = D^{r}M(0).

(ii)

M(θ) = ∑_{r=0}^{∞} \frac{E(X^{r})}{r!}θ^{r}, a < θ < b.

(iii) There is only one distribution function F such that if X has distribution function F then it has moment generating function M.

Proof [sketch] Using the Dominated Convergence Theorem 3.4, we can see that M is infinitely differentiable at 0 with r-derivative equal to the r-moment of X. Moreover, we can see that M is a real analytic function around 0. Hence Taylor’s theorem holds, which yields the second claim. □

EXERCISE 22. Let X be absolutely continuous with density f(x) = π^{−1}(1 + x^{2})^{−1}, x ∈ ℝ. Show that the moment generating function M(θ) is only defined for θ = 0.

3.9 Characteristic functions

Moment generating functions, as the last exercise shows, may not give any information at all. There is an analytic technique bypassing this problem, consisting of replacing the argument of the moment generating function by an imaginary number, i.e. a number of the form it, where i = \sqrt{−1}. First, we define the expectation of a complex random variable X_{1} + iX_{2}, where X_{1}, X_{2} are real random variables, by

E(X_{1} + iX_{2}) := EX_{1} + iEX_{2},

and observe that

|EX| ≤ E|X|.

We then define the characteristic function of X by

φ(t) := Ee^{itX}, t ∈ ℝ.

By Taylor-expansion of the exponential function we can see that

e^{itX} = cos(tX) + i sin(tX).

So |e^{itX}| = cos(tX)^{2} + sin(tX)^{2}| = 1, and therefore φ(t) is defined for all t and |φ(t)| ≤ 1.

Theorem 3.5. Let φ be the characteristic function of a random variable with distribution function F. Suppose that a random variable Y has characteristic function φ. Then Y has distribution function F.

In other words, the charateristic function uniquely determines (characterises) the law of a random variable. It can be shown that:

1. φ is continuous (and, in fact, analytic) function.

2. The complex conjugate φ(t) is φ(−t). If X is symmetric around 0 (i.e. −X has the same law as X) then φ(t) is a real number.

3. It is a positive-semidefinite function, i.e. for every n ∈ ℕ, for every a_{1}, …, a_{n} ∈ ℂ, and every t_{1}, …, t_{n} ∈ ℝ, we have

∑_{k=1}^{n} ∑_{ℓ=1}^{n} a_{k}\overline{a_{ℓ}}φ(t_{k} − t_{ℓ}) ≥ 0.

This property says that if we form the n × n matrix with entries φ(t_{k} − t_{ℓ}), then this matrix has nonnegative eigenvalues. If a random variable X is absolutely continuous with density f, then its characteristic function is given by

φ(t) = ∫_{−∞}^{∞} e^{itx}f(x)dx.

This is the Fourier transform of f. The inverse Fourier transform determines f from φ:

f(x) = \frac{1}{π} ∫_{−∞}^{∞} e^{−itx}φ(t)dt.

EXERCISE 23. Show that the random variable of Exercise 22 has characteristic function φ(t) = e^{−|t|}.

References

[1] B. Fristedt & L. Gray, A Modern Approach to Probability Theory, Birkhäuser, 1997.

[2] E. Hewitt & K. Stromberg, Real and Abstract Analysis, Springer, 1965.

[3] T. Konstantopoulos, Extended set of lecture notes, with proofs,

   www.ma.hw.ac.uk/∼takis

[4] D. Williams, Probability with Martingales, Cambridge, 1991.